Image processing apparatus, image pickup apparatus, control method for image processing apparatus, and storage medium storing control program therefor

ABSTRACT

An image processing apparatus capable of suppressing color fringing in a color image further effectively by image processing. A determination unit determines a region in which signal levels for a color plane in a color image produced by photoelectric conversion of an optical image of a subject exhibit a monotonic increase or a monotonic decrease, as a color fringing region in which color fringing occurs. An estimation unit estimates an intensity of the color fringing in the color fringing region determined by the determination unit. A removal unit deducts an estimate value of the intensity of the color fringing estimated by the estimation unit from the intensity of the color fringing in the color fringing region.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imagepickup apparatus, a control method for the image processing apparatus,and a storage medium storing a control program therefor, which havefunctions for suppressing color fringing in a taken color image.

2. Description of the Related Art

Conventionally, in an image pickup apparatus capable of taking colorimages, chromatic aberration in an imaging optical system that makes anoptical image of a subject be formed on an image pickup device may causea color that does not originally exist to appear around a bright part ofthe image as color fringing. In visible light color imaging using animage pickup apparatus, color fringing easily occurs in a part away fromgreen, which is the central wavelength for the imaging optical system,and an artifact in blue or red, or purple, which is a mixture thereof,appears as a fringe, which is called “color fringe” or “purple fringe”or the like.

Chromatic aberration in an imaging optical system in an image pickupapparatus can be optically suppressed by a combination of plural lenseshaving different dispersions. However, in recent years, as image pickupapparatuses (digital cameras) have been downsized, there is anincreasing need for high resolution for an image sensor used for animage pickup device as well as downsizing of optical systems, which hascaused difficulty in sufficiently suppressing chromatic aberration bythe optical system alone. Therefore, there is a need for suppressing theaforementioned artifact by image processing.

Chromatic aberration is roughly divided into transverse chromaticaberration (magnification chromatic aberration) and longitudinalchromatic aberration (axial chromatic aberration). Transverse chromaticaberration is, as shown in FIG. 17, a phenomenon in which the imagelocation shifts in a direction along the image plane according to thewavelength. Also, longitudinal chromatic aberration is, as shown in FIG.18, is a phenomenon in which the image location shifts in a directionalong the optical axis according to the wavelength. When a primarycolor-type digital imaging system is used, color fringing caused bytransverse chromatic aberration can be corrected by geometric conversionin which different distortions are provided to respective color planesfor R (red), G (green) and B (blue) (see, for example, U.S. Pat. No.6,724,702B1).

Meanwhile, where an image is focused with reference to the plane for G(green), which exhibits the central wavelength of the visible lightrange, longitudinal chromatic aberration causes the image to be unclear(blurred) because the focal points on the planes for R (red) and B(blue), which reside in edges of the visible light range, cannot bebrought to the subject. Color fringing caused by longitudinal chromaticaberration cannot be corrected by such geometric conversion as mentionedabove.

A method for correcting color fringing using the characteristic of colorfringing mainly occurring around a whiteout region (a region reaching apreset signal saturation level) has been proposed (see, for example,Japanese Laid-Open Patent Publication (Kokai) No. 2007-133592). In themethod, a saturated region of the G (green) plane is searched for,signals of pixels surrounding the saturated region are integrated tocalculate the amount of correction, and color fringing is corrected.Also, methods in which the color saturation of a region expected tocause color fringing is lowered to diminish the appearance of colorfringing have been proposed (see, for example, Japanese Laid-Open PatentPublication (Kokai) No. 2001-145117).

Although in an image taken by an image pickup apparatus, color fringingoccurs mainly around a whiteout region, color fringing that gives afeeling of discomfort to viewers occurs also in a region with nowhiteout. For example, in an image taken of a scene of sunbeamsstreaming through the trees, distinct color fringing occurs at theboundary between the blue sky background with no whiteout and the treebranches.

However, a sufficient effect cannot be obtained by the processing ofsearching for a whiteout region and correcting color fringing in itssurrounding region, like that disclosed in Japanese Laid-Open PatentPublication (Kokai) No. 2007-133592, alone. Meanwhile, the processing oflowering the color saturation of a region expected to cause colorfringing, like that described in Japanese Laid-Open Patent Publication(Kokai) No. 2001-145117, provides an effect of eliminating the color incolor fringing to reduce unnaturalness, but may lower the colorsaturation of a region with no color fringing because the processingalso affects the original colors of the subject and a processing-targetregion is determined regardless of whether or not the region actuallyhas color fringing.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus, an imagepickup apparatus, a control method for the image processing apparatus,and a storage medium storing a control program therefor capable ofsuppressing color fringing in a color image further effectively by imageprocessing.

Accordingly, the present invention provides an image processingapparatus comprising a determination unit adapted to determine a regionin which signal levels for a color plane in a color image produced byphotoelectric conversion of an optical image of a subject exhibit amonotonic increase or a monotonic decrease, as a color fringing regionin which color fringing occurs, an estimation unit adapted to estimatean intensity of the color fringing in the color fringing regiondetermined by the determination unit, and a removal unit adapted todeduct an estimate value of the intensity of the color fringingestimated by the estimation unit from the intensity of the colorfringing in the color fringing region.

Accordingly, the present invention provides an image pickup apparatuscomprising the image processing apparatus described above.

Accordingly, the present invention provides A control method for animage processing apparatus comprising a determination step ofdetermining a region in which signal levels for a color plane in a colorimage produced by photoelectric conversion of an optical image of asubject exhibit a monotonic increase or a monotonic decrease, as a colorfringing region in which color fringing occurs, an estimation step ofestimating an intensity of the color fringing in the color fringingregion determined in the determination step, and a removal step ofdeducting an estimate value of the intensity of the color fringingestimated in the estimation step from the intensity of the colorfringing in the color fringing region.

Accordingly, the present invention provides a computer-readable storagemedium storing a control program causing a computer to execute the abovedescribed control method for the image processing apparatus.

According to the present invention, the intensity of color fringing in acolor fringing region in which the signal levels exhibit a monotonicincrease or a monotonic decrease is estimated, and the estimate value ofthe intensity of the color fringing is deducted from the intensity ofthe color fringing in the color fringing region. Consequently, colorfringing in a color image can be further effectively suppressed by imageprocessing.

The features and advantages of the invention will become more apparentfrom the following detailed description taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of a main part of animage pickup apparatus, which is an image processing apparatus accordingto an embodiment of the present invention.

FIG. 2 shows an arrangement of color elements in a primary-color filterincluded in an image pickup device in the image pickup apparatus shownin FIG. 1.

FIG. 3 shows blue fringing occurs at the boundary between bright anddark regions in a color image taken and produced by the image pickupapparatus shown in FIG. 1.

FIG. 4 is a flowchart showing color fringing removal process by imageprocessing in the image pickup apparatus shown in FIG. 1.

FIG. 5 is a flowchart showing monotonic increase/decrease determinationprocess by image processing in the image pickup apparatus shown in FIG.1.

FIG. 6 is a graph showing an example of monotonic increase/decreasedetermination target pixel sections exhibiting a monotonicincrease/decrease characteristic.

FIG. 7 is a graph showing an example of a monotonic increase/decreasedetermination target pixel section exhibiting no monotonicincrease/decrease characteristic.

FIGS. 8A to 8D each show a pixel section for monotonic increase/decreasedetermination, the pixel section having an attention pixel in itscenter, in the monotonic increase/decrease determination process shownin FIG. 5.

FIGS. 9A to 9H each show a pixel section for monotonic increase/decreasedetermination, the pixel section having an attention pixel in an endthereof, in the monotonic increase/decrease determination process shownin FIG. 5.

FIG. 10 shows a region of 3×3 pixels in an image.

FIG. 11 shows the result of applying low-pass filtering process to therespective pixels in a region of 3×3 pixels in an image.

FIG. 12 is a graph showing an example of input signal change whenapplying low-pass filtering process to each pixel.

FIG. 13 is a graph showing typical intensity change in blue fringing.

FIG. 14 is a graph showing a characteristic of non-linear conversion forthe intensity of a B plane.

FIG. 15 shows chromaticity coordinate a-b plane.

FIG. 16 shows a region of 3×3 pixels in a B plane according to theresult of monotonic increase/decrease determination.

FIG. 17 shows a principle of occurrence of transverse chromaticaberration.

FIG. 18 shows a principle of occurrence of longitudinal chromaticaberration.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an embodiment of the present invention will be describedwith reference to the drawings. First, an overview of the embodiment ofthe present invention is provided.

The present embodiment provides an image processing technique thateffectively removes color fringing occurring in a color image taken byan image pickup apparatus and enables reproduction of original colors.In the present embodiment, an image pickup apparatus includes: adetermination unit adapted to determine a color fringing region in acolor image; an estimation unit adapted to estimate the intensity ofcolor fringing in the color fringing region; and a removal unit adaptedto deduct the estimate value from the intensity of the color fringing inthe color fringing region. The image pickup apparatus further includesan excessive removal suppression unit and a space calculation unit.

The determination unit determines a region in which the levels ofsignals in any color plane from among a plurality of color planesincluded in a color image exhibit a monotonic increase or a monotonicdecrease, as a color fringing region in which color fringing occurs.Also, the determination unit applies low-pass filtering process to aremoval amount plane representing an amount of removal for a colorfringing removal target color plane. Also, the determination unitapplies low-pass filtering process to a color plane before making adetermination, and then makes a determination.

Although several methods can be considered for determining a colorfringing region by the determination unit, here, determination is madebased on the characteristics of change in the intensity of colorfringing in a certain pixel section. Color fringing caused by axialchromatic aberration occurs as a result of the image location shiftingin a direction along the optical axis according to the wavelength, andthus, in the case of, for example, blue fringing, the blue plane(hereinafter, the B plane) will be out of focus (i.e., blurred).

Color fringing due to blurring exists over a certain pixel section, andthe intensity of the color fringing in this case has the characteristicof gradually decreasing from a highlighted portion to a shadowed portionof the image. Accordingly, a region where change in the intensity ofcolor fringing in a certain pixel section has the characteristic ofmonotonically increasing or decreasing is determined as a color fringingregion. Regions for which the determination of whether or not to exhibita monotonic increase or a monotonic decrease is made by thedetermination unit include either a horizontal, vertical or obliquepixel section with an attention pixel as its center, or a horizontal,vertical or oblique pixel section with an attention pixel at an endthereof, in a color image.

The estimation unit estimates the intensity of color fringing in thecolor fringing region according to the difference between the signalintensities for the plurality of the color planes included in the colorimage. The removal unit deducts the estimation value of the colorfringing intensity, which has been estimated by the estimation unit,from the intensity of the color fringing in the color fringing region.In other words, the removal unit determines a plane for a color in awavelength range in which chromatic aberration remains in the imagingoptical system used for taking the image as a color fringing removaltarget, and reduces the intensity of the color fringing portion of thecolor plane.

The excessive removal suppression unit can also suppress hue change inthe color image, which has been caused by the removal unit. In thepresent embodiment, the fringing removal target color plane is the Bplane or the red plane (hereinafter, “R plane”). In other words, thefringing removal target color plane color is at least one of the B planeand the R plane. A color-difference plane U/V indicating the color inthe wavelength range in which chromatic aberration remains in theimaging optical system used for taking the color image may be determinedas a color fringing removal target.

Since color fringing is a spatial action, a space calculation isperformed by the space calculation unit in order to estimate the imageintensity of color fringing from the color image. The space calculationunit calculates the gradient of the signal intensity in a color plane.The estimation unit estimates the intensity of the color fringing in thecolor fringing region, using either the gradient of the signal intensityin the color fringing removal target color plane, which has beencalculated by the space calculation unit, or the gradient of the signalintensity in a color plane (reference plane), other than the colorfringing removal target color plane, set as a reference.

In other words, in space calculation, a color plane for which a highresolution has been provided is referred to as a reference plane, otherthan the removal target color plane. A reference plane is a plane for acolor in a wavelength range in which chromatic aberration has favorablybeen corrected in the imaging optical system used for taking the imageor a color plane representing brightness, and is generally a green plane(hereinafter, “G plane”) or a brightness plane (hereinafter, “Y plane”).Although several types of space calculation can be considered,calculation of an image intensity gradient is used here.

An image intensity gradient refers to a signal intensity gradient for areference plane or a removal target color plane. Where an imageintensity gradient is calculated, the estimation unit outputs a valuedepending on the image intensity gradient. In this case, the output maysimply be a value proportional to the image intensity gradient. Asdescribed above, the estimate value of the amount of color fringing tobe removed can be obtained. However, this estimate value is notnecessarily correct, i.e., there may be a shortage or overage. In thecase of a shortage in the estimate value, the color fringing is notremoved, and some amount of color fringing remains.

Meanwhile, in the case of an overage in the estimate value, an excessiveremoval is performed on the color plane, resulting in an inversion ofthe hue of the color fringing. According to the present inventors'experiment, excessive removal for a color plane produces a significantlyunnatural image compared to insufficient removal for a color plane.Therefore, in order to suppress the hue inversion, the image pickupapparatus includes the excessive removal suppression unit as describedabove.

By the excessive removal suppression unit, only pixels in a certaincolor range are determined to be a target for removal by the removalunit, and the color range after the removal is limited to a certainrange. These two certain ranges are the same. In particular, the removaltarget color plane may be a color plane whose intensity is larger thanthe intensity of the color plane referred to. The image processingmethod as described above can provide an image with reduced colorfringing.

Next, the details of the embodiment of the present invention will bedescribed with reference to FIGS. 1 to 17.

FIG. 1 is a block diagram showing a main part of an image pickupapparatus as an image processing apparatus according to the embodimentof the present invention.

In FIG. 1, the image pickup apparatus includes an imaging optical system100, an image pickup device 110, an A/D conversion unit 120, ademosaicking unit 130, a color fringing removal unit 140, a viewcorrection unit 150, a compression unit 160 and a recording unit 170.The color fringing removal unit 140 is an example for providing thedetermination unit, the estimation unit, the removal unit and the spacecalculation unit according to the present invention.

An image of a subject field is formed on the image pickup device 110 viathe imaging optical system 100. The image pickup device 110 isconfigured as a single-plate color image pickup device including commonprimary color filters, and produces a color image consisting of pluralcolor signals by photoelectric conversion of an optical image of asubject. The primary color filters in the image pickup device 110 arethree types of color filters having a main transparent wavelength rangearound 650 nm, 550 nm and 450 nm, respectively, and produce color planescorresponding to the R (red), G (green) and B (blue) bands,respectively.

Although in the present embodiment, the image pickup device 110 includesthe primary color filters for R, G and B, it is not limited to them. Theimage pickup device 110 may include complementary color filters. Wherecomplementary color filters are employed, a color image including colorplanes for R, G and B can also be provided by color conversion process.

The image pickup device 110 configured as a single-plate color imagepickup device has a structure in which three types of color filters (R,G and B) are spatially arranged for respective pixels as shown in FIG.2, and each pixel can obtain the intensity of only a single color plane.Thus, a color mosaic image is output from the image pickup device 110.The A/D conversion unit 120 converts the color mosaic image output fromthe image pickup device 110 as an analog voltage into digital datasuitable for the subsequent image processing.

The demosaicking unit 130 interpolates the color mosaic image, which hasbeen converted into digital data, by an interpolation technique toproduce a color image with R, G and B color information for all thepixels. Although many methods, such as simple linear interpolation and acomplex technique described in “Color filter array recovery using athreshold-based variable number of gradients” (E. Chang, S. Cheung andD. Pan, Proc. SPIE, vol. 3650, pp. 36-43, January 1999), have beenproposed for the interpolation technique, the interpolation in thepresent invention is not limited to a particular interpolationtechnique.

The resolution of the B plane of the color image produced by thedemosaicking unit 130 is lower than the resolution of the G planebecause of chromatic aberration in the imaging optical system 100. Thus,in the boundary between a bright region and a dark region in the colorimage, as shown in FIG. 3, blue blurs, causing an artifact like blueedging (blue color fringing) around the bright region. The abscissa axisin FIG. 3 indicates brightness and darkness, and the ordinate axisindicates image plane intensity (signal intensity). Red blurs where theresolution of the R plane is lower than the resolution of the G plane.

The color fringing removal unit 140 removes the artifact from the colorimage by image processing. The image processing method according to thepresent embodiment, which relates to artifact removal process, will bedescribed in details later. The view correction unit 150, which mainlyperforms image appearance improvement process on the color image,performs image correction such as, for example, tone curve (gamma)correction, color saturation enhancement, hue correction or edgeenhancement. The compression unit 160 compresses the color image thathas been subjected to the image correction by a method such as JPEG toreduce the size of the color image for recording it. The recording unit170 records the processed digital image signals in a storage medium(e.g., flash memory (R)).

The image pickup device 110, the A/D conversion unit 120, thedemosaicking unit 130, the color fringing removal unit 140, the viewcorrection unit 150 and the compression unit 160 in the image pickupapparatus may be formed of individual (separate) devices or may also beformed of a single device. In other words, in reality, a singlemicroprocessor may perform process to be performed by plural componentsof the image pickup apparatus.

Next, an operation for removing color fringing by image processing inthe image pickup apparatus having the above-described configuration willbe described with reference to the flowchart in FIG. 4.

FIG. 4 is a flowchart showing color fringing removal process performedby image processing in the image pickup apparatus.

In FIG. 4, the present process is performed by the color fringingremoval unit 140 in the image pickup apparatus based on a program. Thecolor fringing removal process by the color fringing removal unit 140includes a space calculation step S151, a region determination stepS152, an estimation step S153, an excessive removal suppression stepS154 and a removal step S155. In the color fringing removal process, theB plane from among the color planes (R, G and B planes) is determined asa color fringing removal target, and the G plane is used as a referenceplane.

First, in the space calculation step S151, the color fringing removalunit 140 calculates brightness gradients Blea and Glea for the B planeand the G plane, respectively, according to the below-indicatedformulae. Here, the position of each pixel in the image pickup device110 is represented by two-dimensional coordinates, and the brightness ofa pixel, which is represented by coordinates (x, y), in the B plane andthe & plane is expressed by B(x, y) and G(x, y).

${Blea} = {\left( {\frac{\mathbb{d}B}{\mathbb{d}x},\frac{\mathbb{d}B}{\mathbb{d}y}} \right)\mspace{50mu} = \left( {\frac{{B\left( {{x + 1},y} \right)} - {B\left( {{x - 1},y} \right)}}{2},\frac{{B\left( {x,{y + 1}} \right)} - {B\left( {x,{y - 1}} \right)}}{2}} \right)}$${Glea} = {\left( {\frac{\mathbb{d}G}{\mathbb{d}x},\frac{\mathbb{d}G}{\mathbb{d}y}} \right)\mspace{50mu} = \left( {\frac{{G\left( {{x + 1},y} \right)} - {G\left( {{x - 1},y} \right)}}{2},\frac{{G\left( {x,{y + 1}} \right)} - {G\left( {x,{y - 1}} \right)}}{2}} \right)}$

Here, G(x+1, y) and B(x+1, y) are brightness values (pixel values) ofthe pixel located just to the right of the attention pixel (x, y) in theG plane and the B plane. G(x−1, y) and B(x−1, y) are brightness valuesof the pixel located just to the left of the attention pixel (x, y) inthe G plane and the B plane. G(x, y+1) and B(x, y+1) are brightnessvalues of the pixel just beneath the attention pixel (x, y) in the Gplane and the B plane. G(x, y−1) and B(x, y−1) are brightness values ofthe pixel just above the attention pixel (x, y) in the G plane and the Bplane.

Next, in the region determination step S152, the color fringing removalunit 140 performs monotonic increase/decrease determination for eachpixel in the B plane. The details of the monotonic increase/decreasedetermination will be described using the flowchart in FIG. 5, and FIGS.6 and 7.

In FIG. 5, first, the color fringing removal unit 140 determines whetheror not the ISO sensitivity is high (step S1520). If the ISO sensitivityis high, the process proceeds to the subsequent step S1521, and if theISO sensitivity is not high, the process proceeds to step S1522. Thecolor fringing removal unit 140 analyzes intensity (brightness) changein input signals input from the demosaicking unit 130 for vertical,horizontal and oblique pixel sections each having the attention pixel asits center (step S1522). Furthermore, the color fringing removal unit140 determines whether or not the intensity change in the input signalsin the pixel sections exhibits a monotonic increase/decreasecharacteristic (step S1523).

As a result of the determination, if the intensity change in the inputsignals in the pixel sections does not exhibit a monotonicincrease/decrease characteristic, the color fringing removal unit 140analyzes intensity change in input signals for the vertical, horizontaland oblique pixel sections each having the attention pixel at an endthereof (step S1524). Furthermore, the color fringing removal unit 140determines whether or not the intensity change in the input signals inthe pixel sections exhibits a monotonic increase/decrease characteristic(step S1525).

FIG. 6 shows an example of a monotonic increase/decrease determinationtarget pixel section having a monotonic increase/decreasecharacteristic, and FIG. 7 shows an example of a monotonicincrease/decrease determination target pixel section not having amonotonic increase/decrease characteristic. A pixel section having amonotonic increase/decrease characteristic means that the input signalintensity consecutively increases or consecutively decreases from oneend to the other end of the pixel section. Monotonic increase/decreasedetermination is made for input signals exhibiting intensity changeillustrated in FIGS. 6 and 7. The white squares shown in FIGS. 6 and 7represent an attention pixel.

For an image whose input signal intensity change exhibits a monotonicincrease/decrease characteristic as shown in FIG. 6, the color fringingremoval unit 140, as a result of the analysis at step S1522, determinesthat the image has a monotonic increase/decrease characteristic (stepS1523) because a pixel section, which is subjected to the monotonicincrease/decrease determination, also has a monotonic increase/decreasecharacteristic. Meanwhile, for an image whose input signal changeexhibits the characteristic of increasing and decreasing as illustratedin FIG. 7, the color fringing removal unit 140 determines that the imagehas no monotonic increase/decrease characteristic (step S1523).

Through the above-described determination, if it has been determinedthat the intensity change in the input signals in the pixel section hasa monotonic increase/decrease characteristic (monotonic increasecharacteristic or monotonic decrease characteristic), the color fringingremoval unit 140 sets a monotonic increase/decrease flag (step S1527).Meanwhile, if it has been determined that the intensity change in theinput signals in the pixel section does not have a monotonicincrease/decrease characteristic (monotonic increase characteristic ormonotonic decrease characteristic), the color fringing removal unit 140clears a monotonic increase/decrease flag (step S1526).

The above-described monotonic increase/decrease determination is appliedfor each pixel in the B plane. As a result, the color fringing removalunit 140 assigns “1” to each pixel if a monotonic increase/decrease flagis set for the pixel, and assigns “0” to the pixel if a monotonicincrease/decrease flag is not set, and produces and holds a monotonicincrease/decrease determination result plane for the results, and usesit at the removal step S155. The details of a method for using themonotonic increase/decrease determination result plane will be describedlater.

Next, a method for setting a pixel section in which the monotonicincrease/decrease determination is made for the attention pixel will bedescribed using FIGS. 8A to 8D and 9A to 9H. FIGS. 5A to 5D each show apixel section for the monotonic increase/decrease determination, thepixel section having an attention pixel in its center. FIGS. 9A to 9Heach show a pixel section for the monotonic increase/decreasedetermination, the pixel section having an attention pixel in an endthereof. For a method for setting a pixel section having an attentionpixel at its center, setting a horizontal/vertical pixel section withthe attention pixel at its center, which is shown in FIGS. 8A and 8B,can be considered.

Also, for a method for setting a pixel section having an attention pixelat its center, setting an oblique pixel section with the attention pixelat its center, which is shown in FIGS. 8C and 8D, can be considered. Inother words, an oblique pixel section can be provided with isotropy ifthe pixel section is set to have a distance equivalent to that of thehorizontal/vertical pixel section. In this case, the angle of theoblique pixel section is not limited to 45°, which is shown in thefigures, and any angle may be set. The distance d of the pixel sectionin that case can be calculated according to the following formula.d=x/cos θ,wherein x indicates a distance in the horizontal direction, and θindicates the angle from a horizontal plane.

Meanwhile, for color fringing around a highlighted portion of an imageor color fringing around a shadowed portion of an image, as a result ofbeing affected by saturation or noise, the determination target pixelsection has no monotonic increase/decrease characteristic despite of theexistence of the color fringing and thus, correct determination cannotbe made. In that case, the method of setting a pixel section having theattention pixel at an end thereof, which is as shown in FIGS. 9A to 9H,is effective.

The monotonic increase/decrease determination is made according to themethod as described above, and if it has been determined that at leastany one of pixel sections shown in FIGS. 8A to 8D and 9A to 9H has amonotonic increase/decrease characteristic, the color fringing removalunit 140 determines the attention pixel is a pixel having a monotonicincrease/decrease characteristic.

The present embodiment uses the input signal intensity as a target forthe monotonic increase/decrease determination, brightness gradient maybe used. In that case, color fringing is determined as existing ifchange in the brightness gradient of a certain pixel section exhibitsthe characteristic of inverting only once. It is effective to adjust aproper value for the number of pixels in a pixel section to the fringewidth of the smallest color fringing from among color fringing occurringunder certain imaging conditions of the image pickup apparatus.

The fringe width of the color fringing differs depending on the imagingconditions of the image pickup apparatus (such as the aperture value,the focal length, the focusing accuracy, the focal position in the imageplane, and the coordinates in the image pickup device). Thus, as aresult of adjusting a proper value for the number of pixels in a pixelsection to color fringing with the smallest width from among variouscolor fringing occurring under various types of imaging conditions,color fringing with the smallest width can be detected. For detection ofcolor fringing with the largest width, a pixel section adjusted to colorfringing with the smallest width may also be used.

As a result of making the monotonic increase/decrease determination asdescribed above, color fringing, which is intended to detect, can beextracted. However, depending on the imaging conditions such as a highISO sensitivity, noise is superimposed on the input signals, loweringthe S/N ratio, and as a result, it can be considered that the colorfringing does not have a monotonic increase/decrease characteristic. Inthat case, it is effective to perform filtering for the input signalsvia a digital filter (step S1521) The present embodiment shows the casewhere a low-pass filter (LPF) is used for a digital filter as anexample.

Although several methods for subjecting the input signals to low-passfiltering can be considered, an example in which the filtering isperformed via a low-pass filter with weighting factors of [1, 2, 1] inwhich the weighting factor of the attention pixel is twice those ofadjacent pixels. In a region of 3×3 pixels in an image shown in FIG. 10,where p is the attention pixel, when filtering is performed in ahorizontal direction via the low-pass filter weighting factors of [1, 2,1], the signal p′ of attention pixel can be expressed by the followingformula.p′=(d·1+p·2+e·1)/4Where d, p, and e are signals of the respective pixels d, p, and e.

When a similar calculation is made for the adjacent pixels, the stateshown in FIG. 11 can be obtained. Next, when filtering is performed in avertical direction via the low-pass filter with weighting factors of [1,2, 1], the signal p″ of attention pixel can be expressed by thefollowing formula.p″=(b′·1+p′·2+g′·1)/4

FIG. 12 shows an example of change in input signals when being subjectedto filtering via a low-pass filter. In FIG. 12, the abscissa axisrepresents a cross-section of an image (a pixel section starting from anattention pixel), and the ordinate axis represents the signal intensityin the plane. In the figure, the solid line indicates input signals, thedotted line indicates the case where the input signals are subjected tofiltering via the low-pass filter with weighting factors of [1, 2, 1],and the dashed and dotted line indicates the case where the inputsignals are subjected to filtering via a low-pass filter with weightingfactors of [1, 4, 6, 4, 1]. Here, [1, 4, 6, 4, 1] means that low-passfiltering process is applied by providing a weighting factor for each ofthe pixels adjacent to the attention pixel and pixels outside and nextto the adjacent pixels.

As a result of smoothing the input signals by the low-pass filteringprocess, it becomes possible to highlight the monotonicincrease/decrease characteristic that color fringing originally has. Inthe present embodiment, low-pass filtering process is applied in theorder from the horizontal direction to the vertical direction, thelow-pass filtering process is not limited to this. The low-passfiltering process may be applied in the reversed order from the verticaldirection to the horizontal direction. Also, horizontal and verticallow-pass filtering process can be applied simultaneously by calculatingtwo-dimensional low-pass filter factors.

Referring back to FIG. 4, in the estimation step S153, the colorfringing removal unit 140 estimates the extra intensity of the B plane,which causes color fringing, for each of the pixels in the color image.FIG. 13 shows typical intensity change in blue color fringing. In FIG.13, the abscissa axis represents a cross-section of the image, and theordinate axis represents the intensities of the B plane and the G plane.In FIG. 13, it is assumed that a light source exists at the left end asa high brightness subject exceeding the saturation brightness. For thearea around the light source, which is originally not bright, the bottomof the intensity change profile extends exponentially owing to lightrunning from the light source due to aberration or flare.

Fringing occurs even in the G plane, which is the reference plane, andit extends to a certain degree and but, is small compared to the Bplane, which is the color fringing removal target. Also, the imagepickup device cannot measure an intensity equal to or exceeding acertain saturation level. In such intensity change, when the intensityof the B plane, which is the color fringing removal target, exceeds theintensity of the G plane, which is the reference plane, blue colorfringing occurs.

In the present embodiment, the amount of fringing in the B plane, whichis the color fringing removal target, is estimated from the gradient ofthe intensity change for the B plane. Therefore, the absolute value ofthe brightness gradient Blea of the B plane is multiplied by a factor k₁to obtain a first estimate fringing amount E1.E1=k ₁ =|Blea|

Here, k₁ is a positive value. However, in a region A1 in which the Bplane is saturated, the brightness gradient is 0, and thus, thebrightness gradient before the saturation cannot be obtained.

Therefore, an estimate fringing amount E2 for the region A1 in which theB plane is saturated is estimated from the brightness gradient Glea inthe intensity change in the G plane.E2=k ₂ |Glea|

Here, k₂ is a positive value.

Next, a nonlinear conversion is performed for the intensity of the Bplane to provide a saturation degree S. This nonlinear conversionindicates whether or not the B plane is saturated, and for a region inwhich the intensity of the B plane is saturated, the saturation degree Sis 1, and for a region in which the intensity of the B plane is small,the saturation degrees is 0. Although the saturation degree S may be abinary value of 0 or 1, the saturation degree S may be a valueconsecutively varying from 0 to 1 as shown in FIG. 14.

Then, the color fringing removal unit 140 selects the estimate fringingamount E1 or the estimate fringing amount E2 calculated at theestimation step S153, based on the provided saturation degree S. Inother words, if the saturation degree S is a binary value of 0 or 1, anew estimate fringing amount E is set as:E=E1 (where S=0); andE=E2 (where S=1).Also, if the saturation degree S is a value consecutively varying from 0to 1, a new estimate fringing amount E is set as:E=(1−S)E1+SE2.

Next, at the excessive removal suppression step S154, the color fringingremoval unit 140 corrects the estimate fringing amount E and determinesan actual removal amount E′. The estimate fringing amount (removalamount) estimated at the estimation step S153 is calculated according toa certain model, and does not necessarily agree with the actual fringingamount.

For example, even for light detected by the same B plane, light with awavelength of 450 nm and light with a wavelength of 400 nm exhibitdifferent manners of fringing; however, this is not taken intoconsideration in the estimation step S153. If the estimate fringingamount (removal amount) is much smaller than actual fringing amount, aslight tincture of blue remains even after the removal of the blue colorfringing. Meanwhile, if the estimate fringing amount (removal amount) ismuch larger than actual fringing amount, the B plane may be overlyreduced relative to the gray background, resulting in yellow-green.

In particular, the latter case (the case resulting in yellow-green)provides unnaturalness and gives large discomfort to viewers. Therefore,at the excessive removal suppression step S154, a limitation is imposedto effect fringing removal only within a certain hue range. Thus, first,the color fringing removal unit 140 calculates the chromaticity of eachpixel, which can be determined as follows, relative to the intensitiesof the respective R, G and B planes.

$\begin{pmatrix}x \\y \\z\end{pmatrix} = {\begin{pmatrix}0.41 & 0.36 & 0.18 \\0.21 & 0.75 & 0.07 \\0.02 & 0.12 & 0.95\end{pmatrix}\begin{pmatrix}R \\G \\B\end{pmatrix}}$ a = 5(x − y) b = 2(y − z)

FIG. 15 shows a chromaticity coordinate a-b plane in which a is theabscissa axis and b is the ordinate axis. As shown in the figure, blueresides in the fourth quadrant of the chromaticity coordinate a-b plane,which is hatched (red, yellow and purple reside in the first quadrant,green and white reside in the second quadrant, and blue-green resides inthe third quadrant). The estimate fringing amount E is deducted from theintensity of the B plane, resulting in B=B−E, and thus, the chromaticitymoves in the upper left direction on the chromaticity coordinate a-bplane, as indicated by dotted arrows. The starting point of each arrowis the chromaticity before the removal of the estimate fringing amountE, and the end point is a chromaticity after the removal of the estimatefringing amount E. Accordingly, if the effective hue range is limited toa>0 and b<0, the following conditions are provided:B>0.22R+0.68G; and B>−1.84R+3.30G.

Thus, in the excessive removal suppression step S154, first, an actualremoval amount E′=0 is set for pixels not satisfying the aboveconditional expressions, and such pixels are excluded from the colorfringing removal target. Consequently, the pixels not satisfying theabove conditional expressions do not vary at the color fringing removalstep S155, and thus, the pixel values thereof will not be affected. InFIG. 15, only the fourth quadrant area, which is hatched, is the removaltarget.

Furthermore, also for pixels satisfying the above conditionalexpressions, a removal amount E′ is set as follows:E′=min(E,B−(0.22R+0.68G),B−(−1.84R+3.30G)).The hue change as a result of removal of the removal amount E1, as shownin solid arrows in FIG. 15, remains within the fourth quadrant.

Although in the present embodiment, the hue change is limited by thefourth quadrant in the chromaticity coordinate a-b plane, it may belimited by any angle. In that case, the conditions are:B>r1·G+r2·R; and B>r3−G+r4·R.

Here, r1 to r4 can be calculated according to the below formulae using alimitation angle θ. The hue limitation is defined by two straight linespassing through the origin of the chromaticity coordinate a-b plane, andθ₁ and θ₂ are angles representing the two straight lines.

${r\; 1} = \frac{{{{- 0.358} \cdot \tan}\;\theta_{1}} - 0.596}{{{{- 0.108} \cdot \tan}\;\theta_{1}} - 0.878}$${r\; 2} = \frac{{{0.2 \cdot \tan}\;\theta_{1}} - 0.193}{{{{- 0.108} \cdot \tan}\;\theta_{1}} - 0.878}$${r\; 3} = \frac{{{{- 0.358} \cdot \tan}\;\theta_{2}} - 0.596}{{{{- 0.108} \cdot \tan}\;\theta_{2}} - 0.878}$${r\; 4} = \frac{{{0.2 \cdot \tan}\;\theta_{2}} - 0.193}{{{{- 0.108} \cdot \tan}\;\theta_{2}} - 0.878}$

Consequently, decreasing the B plane beyond the hue limitation range asa result of the removal step S155 can be prevented. The color fringingremoval unit 140 holds the removal amount E′ for the color fringingremoval target color plane, which has been calculated as describedabove, as a removal amount plane, and delivers it to the removal stepS155. The above-described low-pass filtering process is applied to theremoval amount plane. Although in the present embodiment, a simplifieda-b plane has been used for the hue limitation, hue limitation processmay be performed on the u-v plane using a matrix of 3×3 converted fromRGB into YUV.

In the removal step S155, the color fringing removal unit 140 produces anew B plane by deducting the removal amount E′ from the intensity of theB plane. The color fringing removal target is only the pixels eachhaving a monotonic increase/decrease determination flag of “1” providedat the region determination step S152.

Accordingly, if the monotonic increase/decrease determination flag is“1”, the intensity B′ of the new B plane isB′=B−E′; andif the monotonic increase/decrease determination flag is “0”, theintensity B′ of the new B plane isB′=B.The color image with the B plane corrected as described above isdelivered to the view correction unit 150 as an output of the colorfringing removal unit 140.

The case where the monotonic increase/decrease determination flag value,which is the monotonic increase/decrease determination result for acertain a region of 3×3 pixels in the image shown in FIG. 10, is changedbetween adjacent pixels as shown in FIG. 16 will be considered. In sucha case, the removal amount varies at the boundary between pixels (thecase of performing removal between adjacent pixels and the case of notperforming the removal are mixed), and the intensity sharply changes,which may provide an unnatural image and give a feeling of discomfort toviewers. Therefore, in such a case, the method of low-pass filtering theproduced removal amount plane is effective.

Also, smoothing for the boundary portion (monotonic increase/decreasedetermination result plane) may be performed by calculating a gain foreach pixel in a color plane according to the below formulae using themonotonic increase/decrease determination result plane, and multiplyingthe removal amount by the gain.Gain=(aFlag+bFlag+cFlag+dFlag+pFlag+eFlag+fFlag+gFlag+hFlag)/9E=Gain×E′

In the example shown in FIG. 16, the removal amount E″ for a pixel p canbe calculated as follows.0.56=(0+0+1+0+1+1+0+1+1)/9E=0.56×E′

By the method described above, only color fringing can be removedwithout giving a feeling of discomfort.

As described above, according to the present embodiment, the colorfringing removal unit 140 in the image pickup apparatus determines aregion in which the signal levels in any color plane from among aplurality of color planes included in a color image exhibit a monotonicincrease or a monotonic decrease, as a color fringing region in whichcolor fringing occurs. Furthermore, the color fringing removal unit 140estimates the intensity of the color fringing in the color fringingregion according to the difference in signal intensity between theplurality of color planes included the color image, and deducts theestimate value of the estimated intensity of the color fringing from theintensity of the color fringing in the color fringing region.Consequently, the color fringing in the color image can furthereffectively be suppressed by image processing.

Other Embodiments

Although the present embodiment has been described taking aconfiguration in which an image pickup apparatus includes the imagepickup device 110, the A/D conversion unit 120, the demosaicking unit130, the color fringing removal unit 140, the view correction unit 150,the compression unit 160, and the recording unit 170, as an example, theimage pickup apparatus according to the present invention is not limitedto this. An image pickup apparatus may have a configuration including apart or all of the image pickup device 110 to the recording unit 170excluding the color fringing removal unit 140, and an image processingapparatus having the function of the color fringing removal unit 140 maybe provided separately from the image pickup apparatus. In this case, acolor image taken by the image pickup apparatus and stored in arecording medium (e.g., semiconductor memory or magnetic/optical disk)may be input to the image processing apparatus to perform imageprocessing.

It is to be understood that the object of the present invention may alsobe accomplished by supplying a system or an apparatus with a storagemedium in which a program code of software which realizes the functionsof the above described embodiment is stored, and causing a computer (orCPU or MPU) of the system or apparatus to read out and execute theprogram code stored in the storage medium.

In this case, the program code itself read from the storage mediumrealizes the functions of any of the embodiments described above, andhence the program code and the storage medium in which the program codeis stored constitute the present invention.

Examples of the storage medium for supplying the program code include afloppy (registered trademark) disk, a hard disk, a magnetic-opticaldisk, a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, aDVD+RW, a magnetic tape, a nonvolatile memory card, and a ROM.Alternatively, the program code may be downloaded via a network.

Further, it is to be understood that the functions of the abovedescribed embodiment may be accomplished not only by executing a programcode read out by a computer, but also by causing an OS (operatingsystem) or the like which operates on the computer to perform a part orall of the actual operations based on instructions of the program code.

Further, it is to be understood that the functions of the abovedescribed embodiment may be accomplished by writing a program code readout from the storage medium into a memory provided on an expansion boardinserted into a computer or in an expansion unit connected to thecomputer and then causing a CPU or the like provided in the expansionboard or the expansion unit to perform a part or all of the actualoperations based on instructions of the program code.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims priority from Japanese Patent Application No.2008-118500, filed on Apr. 30, 2008, which is hereby incorporated byreference herein in its entirety.

1. An image processing apparatus comprising: a determination unitadapted to determine a region in which signal levels for a color planein a color image produced by photoelectric conversion of an opticalimage of a subject exhibit a monotonic increase or a monotonic decrease,as a color fringing region in which color fringing occurs; an estimationunit adapted to estimate an intensity of the color fringing in the colorfringing region determined by said determination unit; and a removalunit adapted to deduct an estimate value of the intensity of the colorfringing estimated by said estimation unit from the intensity of thecolor fringing in the color fringing region.
 2. The image processingapparatus according to claim 1, wherein a region for which thedetermination of whether or not signal levels exhibit a monotonicincrease or a monotonic decrease is made by said determination unitincludes either a horizontal, vertical or oblique pixel section with anattention pixel at a center thereof, or a horizontal, vertical oroblique pixel section with an attention pixel at an end thereof, in acolor image.
 3. The image processing apparatus according to claim 1,wherein said estimation unit estimates the intensity of the colorfringing in the color fringing region according to a difference insignal intensity between a plurality of color planes included in thecolor image.
 4. The image processing apparatus according to claim 1,further comprising a space calculation unit adapted to calculate asignal intensity gradient for each of the plurality of color planesincluded in the color image, wherein said estimation unit estimates theintensity of the color fringing in the color fringing region usingeither the signal intensity gradient, calculated by said spacecalculation unit, of a color fringing removal target color plane, or thesignal intensity gradient of a color plane, other than the colorfringing removal target color plane, the color plane being set as areference.
 5. The image processing apparatus according to claim 1,wherein said removal unit determines a plane for a color in a wavelengthrange in which chromatic aberration in an imaging optical system usedfor taking the color image remains, as a color fringing removal target.6. The image processing apparatus according to claim 5, wherein thecolor plane determined by said removal unit as a color fringing removaltarget is at least one of a blue plane and a red plane.
 7. The imageprocessing apparatus according to claim 1, wherein said removal unitdetermines a color-difference plane representing a color in a wavelengthrange in which chromatic aberration remains in an imaging optical systemused for taking the color image remains, as a color fringing removaltarget.
 8. The image processing apparatus according to claim 1, whereinsaid determination unit makes determination for a color plane afterapplying low-pass filtering process to the color plane before makingdetermination.
 9. An image pickup apparatus comprising the imageprocessing apparatus according to claim
 1. 10. The image processingapparatus according to claim 1, wherein said removal unit holds aremoval amount for a color fringing removal target color plane in aremoval amount plane.
 11. The image processing apparatus according toclaim 10, wherein low-pass filtering process is applied to the removalamount plane.
 12. The image processing apparatus according to claim 1,wherein said determination unit produces and holds a monotonicincrease/decrease determination result plane based on a monotonicincrease/decrease determination flag indicating whether or not signallevels for a color plane exhibit a monotonic increase characteristic ora monotonic decrease characteristic.
 13. The image processing apparatusaccording to claim 12, wherein smoothing process is applied to themonotonic increase/decrease determination result plane.
 14. A controlmethod for an image processing apparatus, comprising: a determinationstep of determining a region in which signal levels for a color plane ina color image produced by photoelectric conversion of an optical imageof a subject exhibit a monotonic increase or a monotonic decrease, as acolor fringing region in which color fringing occurs; an estimation stepof estimating an intensity of the color fringing in the color fringingregion determined in said determination step; and a removal step ofdeducting an estimate value of the intensity of the color fringingestimated in said estimation step from the intensity of the colorfringing in the color fringing region.
 15. A non-transitorycomputer-readable storage medium storing a control program causing acomputer to execute the control method for the image processingapparatus according to claim 14.